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ABSTRACT 

We present the results of a study of a large sample of luminous {z' AB < 26) Lyman break 
galaxies (LBGs) in the redshift interval 4.7 < z < 6.3, selected from a contiguous 0.63 
square degree area covered by the UKIDSS Ultra Deep Survey (UDS) and the Subaru XMM- 
Newton Survey (SXDS). Utilising the large area coverage and the excellent available opti- 
cal+nearlR data, we use a photometric redshift analysis to derive a new, robust, measurement 
of the bright end (L > L*) of the UV-selected luminosity function at high redshift. When com- 
bined with literature studies of the fainter LBG population, our new sample provides improved 
constraints on the luminosity function of redshift 5 < z < 6 LBGs over the luminosity range 
0.1L*< L <10L*. A maximum likelihood analysis returns best-fitting Schechter function pa- 
rameters of M* 500 = -20.73i0.ll, 4>* = 0.0009i0.0002Mpc -3 anda = -1.66i0.06for 
the luminosity function at z = 5, mdMf 500 = -20.04 i0.12, <\>* = 0.0018i0.0005 MpcT 3 
and a = — 1.71 i 0.11 at z = 6. In addition, an analysis of the angular clustering proper- 
ties of our LBG sample demonstrates that luminous 5 < z < 6 LBGs are strongly clustered 
(ro = 8.1^^5/i 70 1 Mpc), and are consistent with the occupation of dark matter halos with 
masses of ~ io 2 M . Moreover, by stacking the available multi-wavelength imaging 
data for the high-redshift LBGs it is possible to place useful constraints on their typical stellar 
mass. The results of this analysis suggest that luminous LBGs at 5 < z < 6 have an aver- 
age stellar mass of log 10 (M/ M©) = IO.O^q ^, consistent with the results of the clustering 
analysis assuming plausible values for the ratio of stellar to dark matter. Finally, by combin- 
ing our luminosity function results with those of the stacking analysis we derive estimates of 
~ 1 x 10 7 M Q Mpc~ 3 and ~ 4 x 10 6 M Mpc~ 3 for the stellar mass density at z ~ 5 and 
z ~ 6 respectively. 
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1 INTRODUCTION 

Accurately determining the global properties of the high-redshift 
galaxy population is a powerful method of constraining current 
models of galaxy evolution, and identifying the sources responsible 
for reionisation. Large, statistical samples of high-redshift galax- 
ies can be identified efficiently using two complementary photo- 
metric techniques. Firstly, Lyman-alpha emitters (LAEs) can be se- 
lected via deep imaging with narrow-band filters centred on the red- 
shifted Lyman-alpha emission line (Hu, McMahon & Cowie 1999). 
Alternatively, so-called Lyman-break galaxies (LBGs) can be se- 
lected from deep broad-band photometry using the Lyman-break, 
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or "dropout", technique pioneered by Guhathakurta, Tyson & Ma- 
jewski (1990). 

When combined with deep multi-wavelength photometry and 
spectroscopic follow-up, narrow-band selection of LAEs is an effi- 
cient technique for producing large samples of high-redshift galax- 
ies within narrow redshift intervals, free from significant levels of 
contamination. By exploiting the wide-field imaging capabilities of 
Suprime-Cam on Subaru, several recent studies have investigated 
the number densities, luminosity functions and clustering proper- 
ties of LAEs at z = 5.7 and z = 6.6 (e.g. Ouchi et al. 2008; Shi- 
masaku et al. 2006; Taniguchi et al. 2005). Moreover, at present, the 
highest-redshift galaxy with spectroscopic confirmation (z — 6.96; 
lye et al. 2006) was identified using the narrow-band technique (see 
Stark et al. 2007a for candidate LAEs at z > 8). However, although 
LAE studies have many advantages, they are constrained by the fact 
that any individual study can only survey a comparatively small 
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cosmological volume and that only a minority of LBGs at high 
redshift appear to be strong LAEs (e.g. Shapley et al. 2003). There- 
fore, although the selection function of the Lyman-break technique 
is more difficult to quantify, because it includes both LAEs and 
non-LAEs, it should provide a more complete census of the high- 
redshift galaxy population. 

Consequently, many studies in the recent literature have ex- 
ploited both ground-based and Hubble Space Telescope (HST) data 
to study the properties of high-redshift galaxies selected via the 
Lyman-break technique. In particular, due to its unparallelled point- 
source sensitivity, deep HST imaging has allowed studies of the 
high-redshift luminosity function to reach <0.1L*, and has there- 
fore been crucial to constraining both the normalization and faint- 
end slope of the luminosity function at 3 < z < 6 (e.g. Dickinson 
et al. 2004; Malhotra et al. 2005; Oesch et al. 2007; Bouwens et 
al. 2006, 2007, 2008; Reddy et al. 2007). In terms of best-fitting 
Schechter function parameters the results of these studies have de- 
scribed a fairly consistent picture, with the majority finding a very 
steep faint-end slope, typically in the range —1.5 < a < —1.8 
(although see Sawicki & Thompson 2006 for a different result at 
z ~ 4), and little evidence for significant evolution in either the 
faint-end slope or normalization at z > 3. 

Although studies based on deep HST imaging have been suc- 
cessful at exploring the faint end of the high-redshift luminosity 
function, their small areal coverage means that they are not ide- 
ally suited to accurately constraining the bright end of the luminos- 
ity function. To address this issue studies based on shallower, but 
wider area, ground-based imaging have been important (e.g. Ouchi 
et al. 2004a; Shimasaku et al. 2005; Yoshida et al. 2006; Iwata et 
al. 2007). To date, although the existing wide-area, ground-based 
optical imaging surveys have been very useful for constraining the 
bright end of the LBG luminosity function in the redshift range 
3 < z < 5, it has been difficult to successfully extend ground- 
based studies to higher redshift. The reason for this is that without 
the benefits of HST spatial resolution to exclude low-redshift con- 
taminants (i.e red galaxies at z ~ 1 and ultra-cool galactic stars), 
selecting clean samples of z > 5 LBG candidates requires near- 
infrared photometry] to confirm a relatively blue spectral index 
long-ward of Lyman alpha. Unfortunately, due to the lack of wide 
area near-infrared detectors, until very recently it was not possible 
to obtain suitably deep near-infrared imaging over areas commen- 
surate with the existing wide-area optical surveys. 

However, with the advent of deep near-infrared imaging over 
a 0.8 square degree field provided by the UKIDSS Ultra Deep Sur- 
vey (UDS) this fundamental issue is now being addressed. In a 
previous paper (McLure et al. 2006) we combined the early data 
release of the UDS with optical imaging from the Subaru/XMM- 
Newton survey (SXDS) to explore the properties of the most lu- 
minous (z' < 25) galaxies at z > 5. In this paper we extend this 
previous study by using new, deeper, JK imaging of the UDS field 
to study the properties of a much larger sample of z < 26 LBGs 
in the redshift range 4.7 < z < 6.3. The specific aim of this study 
was to use the large, contiguous area of the UDS to provide an 
improved measurement of the bright end of the UV-selected lumi- 
nosity function at z = 5 and z = 6, and to measure the clustering 
properties, and thereby halo masses, of the luminous LBG popula- 
tion at z > 5. 



The use of two medium-band z— filters to quantify the UV slope of a 
small sample of 12 redshift 5.6 < z < 6.2 candidates in the Subaru Deep 
Field by Shimasaku et al. (2005) is one notable exception. 



The structure of the paper is as follows. In Section 2 we briefly 
describe the properties of the near-infrared and optical data-sets 
used in this study. In Section 3 we describe our initial selection 
criteria and our photometric redshift analysis. In Section 4 we de- 
scribe our adopted technique for estimating the LBG luminosity 
function. In Section 5 we present our luminosity function results, 
and the results of combining our ground-based data with existing 
constraints derived from deep HST imaging data. In Section 6 we 
report the results of our study of the clustering properties of the 
luminous LBG population in the redshift range 5 < z < 6 and 
link the LBG population to that of the underlying dark matter ha- 
los. In Section 7 we provide an estimate of the typical stellar mass 
for luminous 5 < z < 6 LBGs based on a stacking analysis of 
the available multi-wavelength imaging data. In Section 8 we use 
this information to derive an estimate of the stellar mass function 
at z = 5 and z — 6 and compare our results with the predictions of 
recent semi-analytic models of galaxy formation. In Section 9 we 
summarise our main conclusions. Throughout the paper we adopt 
the following cosmology: Ho = 70 km s~ Mpc~\ Q, m = 0.3, 
SIa = 0.7, n s — 1.0, as = 0.9. All magnitudes are quoted in the 
AB system (Oke & Gunn 1983). 



2 THE DATA 

The Ultra Deep Survey (UDS) is the deepest of five near- 
infrared surveys currently underway at the UK InfraRed Telescope 
(UKIRT) with the new WFCAM imager (Casali et al. 2007) which 
together comprise the UKIRT Infrared Deep Sky Survey (UKIDSS; 
Lawrence et al. 2007). The UDS covers an area of 0.8 square de- 
grees centred on RA=02: 17:48, Dec=-05:05:57 (J2000) and is al- 
ready the deepest, large area, near-infrared survey ever undertaken. 
The data utilized in this paper were taken from the first UKIDSS 
data release (DR1; Warren et al. 2007), which included JK imag- 
ing of the entire UDS field to 5a depths of J = 23.9, K = 
23.8 (1.6" diameter apertures). The UKIDSS DR1 became publicly 
available to the world-wide astronomical community in January 
2008 and can be downloaded from the WFCAM Science Archive. 


The UDS field is covered by a wide variety of deep, multi- 
wavelength observations ranging from the X-ray through to the 
radio (see Cirasuolo et al. 2008 for a recent summary). However, 
for the present study the most important multi-wavelength observa- 
tions are the deep optical imaging of the field taken with Suprime- 
Cam (Miyazaki et al. 2002) on Subaru as part of the Subaru/XMM- 
Newton Deep Survey (Sekiguchi et al. 2005). The optical imag- 
ing consists of 5 over-lapping Suprime-Cam pointings, and covers 
an area of ~ 1.3 square degrees. The whole field has been im- 
aged in the BVRi'z' filters, to typical 5a depths of B = 27.9, 
V = 27.4, R = 27.2, i' = 27.2 and z = 26.2 (1.6" diameter aper- 
tures). The reduced optical imaging of the SXDS is now publicly 
available and full details of the observations, data reduction and 
calibration procedures are provided in Furusawa et al. (2008). The 
high-redshift galaxies investigated in this study were selected from 
a contiguous area of 0.63 square degrees (excluding areas contami- 
nated by bright stars and CCD blooming) covered by both the UDS 
near-infrared and SXDS optical imaging. 



2 http://surveys.roe.ac.uk/wsa/ 

3 http://www.naoj.org/Science/SubaruProject/SDS/ 
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the only other selection criterion applied was the exclusion of all 
objects which were detected in the B— band at more than 2a sig- 
nificance. This criterion was adopted in order to exclude the vast 
majority of galaxies which are simply too bright in the B— band 
to be robust high-redshift candidates; given that the Lyman-limit 
at 912A is redshifted out of the Suprime-cam B— band filter at a 
redshift of 2 = 4.5. 

Following the application of the initial selection criteria the 
parent sample of potential high-redshift galaxies consisted of 6495 
candidates in the apparent z'— band magnitude range 23.8 < z < 
26.0. Due to the decision not to apply strict drop-out criteria in our 
initial selection, in principle, this sample should contain all galaxies 
with z < 26 lying between z = 4.5 (when the Lyman limit [912A] 
is redshifted out of the B— band) and z = 7.0 (when Lyman alpha 
[1216A] is redshifted out of the z'-band). However, given that the 
primary selection is performed in the rest-frame UV, it is clear that 
the sample is still biased against objects with significant levels of 
intrinsic reddening. 



M 1500 ( AB ) 

Figure 1. The V/V m ax estimates of the bright end of the UV-selected lu- 
minosity function in two redshift intervals centred on z = 5 & 2 = 6. The 
absolute UV magnitudes have been calculated at a rest-frame wavelength 
of 15()0A. The horizontal error bars indicate the width of the magnitude 
bins adopted (either Am = 0.25 or Am = 0.5 depending on signal-to- 
noise). The vertical error bars indicate the uncertainty due to simple poisson 
statistics. The faintest bin in the luminosity function estimate is set by our 
adopted magnitude limit of 2' = 26. The bright limit is set by the require- 
ment that the brightest bin should contain the equivalent of more than one 
object. 



3 HIGH REDSHIFT CANDIDATE SELECTION 

One of the principle motivations for this study was to exploit 
the unique combination of areal coverage and optical+nearlR data 
available in the UDS/SXDS to study the high-redshift galaxy pop- 
ulation without recourse to strict dropout/LBG selection criteria. 
The reasoning behind this is to allow the luminosity function to 
be studied over a relatively wide range of redshifts from a single 
sample, and to reduce as much as possible the strong bias towards 
young, blue star-forming galaxies inherent to traditional Lyman- 
break selection. Although single-colour selection techniques (e.g. 
V-drop, i— drop) have proven to be highly successful at isolating 
high-redshift galaxies, because of the need to adopt fairly strict 
colour-cut criteria (e.g. i — z > 1.3 for 2 > 5.5), it is at least 
possible that a significant population of old/redder, perhaps more 
massive, galaxies which marginally fail to satisfy these strict cri- 
teria could be excluded (e.g. Dunlop et al. 2007; Rodighiero et al. 
2007). 

Consequently, from the outset the decision was taken to make 
the initial selection criteria as simple and inclusive as possible, and 
to then use a photometric redshift analysis to both define the final 
high-redshift galaxy sample and to exclude likely low-redshift in- 
terlopers. Within this context, the original sample for this study was 
a 2'— band selected catalog covering the 0.63 square-degree area 
uniformly covered by the Subaru optical and UDS near-infrared 
data, produced with version 2.5.2 of the SEXTRACTOR software 
(Bertin & Arnouts 1996). The original catalog was then reduced 
to ~ 300, 000 objects by adopting a 2'— band magnitude limit of 
2' = 26, which corresponds to the ~ 6a detection threshold (1.6" 
diameter apertures) and the 80% completeness limit. In addition, 



3.1 Photometric redshift analysis 

The next stage in the analysis was to process each of the po- 
tential high-redshift candidates with our own photometric redshift 
code. The photometric redshift code is an extended version of the 
publicly available HYPERZ package (Bolzonella, Miralles & Pello 
2000) and is described in more detail by Cirasuolo et al. (2007). 
However, briefly, the code fits a wide range of different galaxy 
SED templates to the available multi-wavelength (BVRi'z'JK) 
photometry of each candidate, returning a best-fitting value of red- 
shift, SED type, age, mass and reddening. It is worth noting at this 
point that for calculating the photometric redshifts, and the final 
luminosity function estimates, the 1.6" diameter magnitudes were 
corrected to total using point-source corrections in the range 0.21- 
0.30 magnitudes (depending on the seeing of the BVRi'z'JK 
images). The primary set of stellar population models we adopted 
were those of Bruzual & Chariot (2003), with solar metallicity and 
assuming a Salpeter initial mass function (IMF) with a lower and 
upper mass cutoff of 0.1 and 100 Mq respectively. Both instan- 
taneous burst and exponentially declining star formation models - 
with e-folding times in the range 0.1 < r(Gyr) < 15 - were in- 
cluded, with the restriction that the SED models did not exceed the 
age of the Universe at the best-fitting redshift. The code accounts 
for dust reddening by following the Calzetti et al. (2000) obscura- 
tion law within the range < Ay < 6, and accounts for Lyman 
series absorption due to the HI clouds in the inter galactic medium 
according to the Madau (1995) prescription. 

Based on the results of the photometric redshift analysis, 8% 
of the sample was excluded because it was not possible to find an 
acceptable \ 2 fit (at the 3a level) with a galaxy SED template at any 
redshift. The vast majority of these objects were spurious artifacts 
on the 2'-band images which had contaminated the original catalog. 
However > 50% of the excluded contaminants with 2' < 24.5 had 
2' — J colours consistent with ultra-cool galactic stars. 

3.2 Multiple redshift minima 

Following the photometric redshift analysis our sample consisted 
of 1621 high-redshift galaxy candidates with primary redshift solu- 
tions at 2 > 4.5, and 4350 objects with primary redshift solutions at 
2 < 4.5. However, due to the problem of multiple redshift minima 
which is inherent to photometric redshift techniques, when com- 
puting the luminosity function is it clearly not optimal to simply 
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exclude the majority of candidates with primary redshift solutions 
at z < 4.5. Indeed, even for candidates which have well defined 
primary redshift solutions at z > 4.5, it is often the case that a 
plausible secondary redshift solution exists at lower redshift; and 
vice- versa. The simple reason is that, within the constraints of the 
available photometry, it is often statistically impossible to cleanly 
differentiate between a high-redshift Lyman-break solution and a 
competing, low-redshift, 4000 A -break solution at z ~ 1. There- 
fore, during the derivation of the luminosity functions presented in 
Section 5, each candidate (including those with primary redshift 
solutions at z < 4.5) was represented by its normalised probability 
density function P{z), which satisfied the condition: 

/ P(z')Sz' = 1 (1) 

In addition to making better use of the available information, this 
methodology immediately deals with the problem of multiple red- 
shift solutions in a natural and transparent fashion. 



4 LUMINOSITY FUNCTION ESTIMATION 

Armed with a normalised probability density function for each 
high-redshift candidate, it is possible to construct their luminosity 
function using the classic V/Vmax maximum likelihood estima- 
tor of Schmidt (1968). Therefore, within a given redshift interval 
(zmin < z < z ma x), the luminosity function estimate within a 
given absolute magnitude bin was calculated as follows: 

where the summation runs over the full sample, and z\ — > zi is the 
redshift range within which a candidate's absolute magnitude lies 
within the magnitude bin in question. The quantity Ciirrii, z) is a 
correction factor which accounts for incompleteness, object blend- 
ing and contamination from objects photometrically scattered into 
the sample from faint-ward of the 2'— band magnitude limit0. 

4.1 Completeness and object blending 

The corrections for sample incompleteness and object blending 
were calculated by adding thousands of randomly placed point- 
sources into the z — band images and recovering them with SEX- 
TRACTOR using the same extraction parameters adopted for the 
original object catalogues. The results of this procedure revealed 
that the z'— band images are 100% complete at z 1 < 25 and fall 
to 80% complete at the adopted z' = 26 magnitude limit. In addi- 
tion, this process also demonstrated that it is necessary to correct 
for a constant ~ 15% of objects which are missed from the ob- 
ject catalogues due to blending with nearby objects on the crowded 
Suprime-Cam images. 

4.2 Photometric scattering 

In this study we adopt z = 26 as our limiting magnitude, which 
corresponds to the 6a detection limit. This fact, combined with the 
steepness of the luminosity function in the region we are exploring 

4 A full discussion of the influence on the estimated luminosity functions 
of adopting the full photometric redshift probability density function for 
each LBG candidate is provided in the appendix. 



(see Fig. 1) means that the prospect of contamination of the faint 
end of our sample from objects photometrically up-scattered from 
below the flux limit has to be considered carefully. In order to quan- 
tify this effect it is necessary to rely on simulation. In outline, the 
adopted procedure was to produce a realistic simulated population 
of LBGs in the redshift interval 4.0 < z < 6.5, based on a known 
model for the evolving luminosity function. The simulated photom- 
etry for this population was then passed through the same initial 
selection criteria and photometric redshift analysis as employed on 
the real data. The average effect on the faint end of our luminosity 
function determination could then be calculated by running many 
monte-carlo realisations of the simulated LBG population. 

In order to make the simulated LBG population as realistic as 
possible we based our model for the evolving luminosity function 
on the Schechter function fits to the latest determination of the UV- 
selected luminosity functions centred on z — 3.8, z = 5.0 and 
z — 5.9 by Bouwens et al. (2007). Over the full redshift range the 
adopted model had a fixed normalisation ((f)* = 0.0015 Mpc -3 ), 
a fixed faint-end slope (a = —1.75) and an evolving characteristic 
magnitude parameterised as: 

AfJeoo(*) = -21.0 + 1.35 logfz - 2.8) (3) 

where M* 50u is calculated at a rest-frame wavelength of 1500A 
(which corresponds to the middle of the z'— filter for an object at 
z = 5). Although simple, this parameterisation nonetheless closely 
reproduces the Bouwens et al. (2007) luminosity function fits over 
the redshift range of interest. More importantly, this model should 
provide an accurate prediction of the relative numbers of objects 
just above and just below our adopted magnitude limit, the region 
in which our sample is most vulnerable to potential contamination 
from photometric scatter. 

This parameterisation of the evolving luminosity function was 
used to populate the redshift— magnitude plane using bin sizes of 
Az = 0.1 and Am = 0.25. Within each redshift bin the lumi- 
nosity function was integrated down to an absolute magnitude cor- 
responding to z — 27, one magnitude fainter than our adopted 
magnitude limit. In addition to its absolute magnitude and redshift, 
each simulated LBG was allocated BVRi'z'JK photometry based 
on an appropriate model SED drawn randomly from a distribution 
of parameter values (i.e. age, reddening, metallicity) representative 
of those displayed by the real LBG sample. Appropriate photomet- 
ric errors were then calculated for the synthetic photometry in each 
filter, according to the average depths of the real data. 

The final stage of the process involved the use of monte-carlo 
simulation whereby, in each realisation, the photometry of each 
candidate was randomly perturbed according to its photometric er- 
rors. The perturbed photometry of the synthetic LBG sample was 
then passed through our initial optical selection criteria to estab- 
lish the number of objects which would have been included in our 
sample. The final result of this process was a finely sampled grid 
of correction factors which accounted for the fraction of the objects 
scattered into, and out of, the sample as a function of redshift and 
apparent z'— band magnitude. 

In terms of the estimated luminosity functions presented in 
Section 5, this simulation process demonstrated that photometric 
up-scatter from below the magnitude limit has a relatively minor 
effect. As would be anticipated, the effect is most pronounced on 
the two faintest absolute magnitude bins in the 2 = 5 and z = 6 
luminosity function estimates. However, even here the effect is only 
to increase the volume density by ~ 25% and ~ 15% respectively. 
The luminosity function estimates shown in Fig. 1 are corrected for 
this effect. 
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Figure 2. A comparison of our estimates of the bright end of the luminosity function at z = 5 and 2 = 6 with the Schechter function fits derived by recent 
literature studies. The left-hand panel compares our luminosity function estimate at 2 = 5 with the results of Iwata et al. (2007), Yoshida et al. (2006) and 
Bouwens et al. (2007). The right-hand panel compares our luminosity function estimate at 2 = 6 with the results of Shimasaku et al. (2005) and Bouwens et 
al. (2007). In each panel the literature luminosity function fits are plotted as solid curves where they are well constrained by data, and as dotted curves where 
there is no data, or only data with very large uncertainties. 



5 THE LUMINOSITY FUNCTION AT HIGH REDSHIFT 

Following the procedures outlined in the previous two sections, the 
UV-selected luminosity functions were estimated in two redshift 
intervals centred on 2 = 5 and z = 6. For the z = 5 lumi- 
nosity function we considered the redshift range 4.7 < z < 5.3 
and for the 2 = 6 luminosity function we considered the redshift 
range 5.7 < z < 6.3. These redshift ranges were chosen to be 
wide enough to include sufficient numbers of objects to provide 
a robust measurement of the bright end of the luminosity function, 
and narrow enough to limit the evolution of the luminosity function 
across each bin. The resulting estimates of the luminosity functions 
at 2 = 5 and 2 = 6 are shown in Fig. 1. 

The advantages provided by the large area and deep opti- 
cal+nearlR data available in the UDS/SXDS are immediately ob- 
vious from Fig. 1. As a result of the current data-set covering an 
area of 0.63 square degrees, it has been possible to robustly esti- 
mate the bright end of the luminosity function to absolute magni- 
tudes as bright as M1500 — —22.5 and M1500 — —22 at 2 = 5 
and 2 = 6 respectively. A second obvious feature of Fig. 1 is the 
significant evolution in the bright end of the luminosity function 
within the 5 < 2 < 6 redshift interval; as previously demonstrated 
by Bouwens et al. (2007). From the data presented in Fig. 1 it is 
tempting to assume that this evolution is due entirely to a change in 
Mi 500 between 2 = 5 and 2 = 6 (a shift of AM* 500 ~ 0.5 mag- 
nitudes results in excellent agreement between the two luminosity 
functions). In fact, based on our data-set alone, it is not possible 
to determine whether the apparent evolution is due entirely to a 
change in M1500, or whether significant evolution in the normal- 
isation (((>*) is also occurring. However, the differential change in 
galaxy number density as a function of absolute magnitude does 
provide some information on the form of the luminosity function 
evolution. For example, while the number density of M1500 — — 21 
galaxies only decreases by a factor of ~ 3 between 2 = 5 and 
2 = 6, the number density of M1500 — —22 galaxies decreases by 



a factor of ~ 10. In terms of Schechter function parameters, it is 
not possible to reproduce this change in luminosity function shape 
through evolution of (f>* alone, and immediately confirms that some 
evolution of M* 500 must be taking place. An attempt to quantify 
the relative contribution of evolution in M* 500 and (f>* is pursued in 
Section 5.2. 

5.1 Comparison with previous studies 

Before proceeding to explore the evolution of the luminosity func- 
tion from 2 = 5 to 2 = 6 further, it is worthwhile comparing our 
results with those of recent studies in the literature. 

5.1.1 The z ~ 5 luminosity function 

The 2 ~ 5 luminosity function has been investigated by several re- 
cent studies using both ground-based and HST imaging (e.g. Ouchi 
et al. 2004; Yoshida et al. 2006; Iwata et al. 2007; Oesch et al. 
2007; Bouwens et al. 2007). As an illustration of the range of re- 
sults which have recently been reported we show in Fig. 2a a com- 
parison of our 2 = 5 luminosity function estimate with the ground- 
based results of Yoshida et al. (2006) and Iwata et al. (2007), and 
the HST-based results of Bouwens et al. (2007). The Yoshida et al. 
(2006) study is based on deep Subaru Suprime-cam optical imag- 
ing data covering an area of ~ 0.25 square degrees in the Subaru 
Deep Field. Yoshida et al. selected a large LBG sample centred on 
2 = 4.7, based on cuts in the V — i' vs i' — z colour-colour dia- 
gram, the reliability of which were confirmed via limited follow-up 
spectroscopy. The Schechter function fit derived by Yoshida et al. 
from their 2 = 4.7 LBG sample is shown as the dark grey curve 
in Fig. 2a. It can be seen that our luminosity function estimate 
is in excellent agreement with the Yoshida et al. determination at 
A^i500 < —21.5. At M1500 > —21.5 our number densities are 
somewhat lower than determined by Yoshida et al., although only 
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Figure 3. A comparison of our luminosity function estimates at z = 5 and z = 6 with the equivalent estimates derived from deep HST imaging by Bouwens 
et al. (2007). It can be seen that the agreement between the two independent luminosity function estimates at both z = 5 and z = 6 is excellent. This figure 
illustrates the advantages provided by the wide area coverage of the current UDS/SXDS study. Although the unparallelled sensitivity of HST imaging provides 
tight constraints on the normalisation and faint-end slope of the z > 5 luminosity function, the wide area of the current UDS/SXDS study provides improved 
constraints on the bright end of the the luminosity function, and extends the dynamic range of the data by a factor of ~ 3 in luminosity. 



by ~ 20%. This difference is not very significant, and can largely 
be explained by the fact that the Schechter function fit derived by 
Yoshida et al. does not have a well constrained faint-end slope due 
to the lack of data fainter than ~ Mi 50 o. We note here that the 
Yoshida et al. results at z ~ 5 are very similar to those of Ouchi et 
al. (2004a), who analysed an combined SDF+SXDS data-set cov- 
ering a larger area (0.35 square degrees) but with a magnitude limit 
~ 0.5 magnitudes brighter. 

The solid black curve plotted in Fig. 2a is the Schechter func- 
tion fit derived by Bouwens et al. (2007) from a sample of 1416 
V— drop galaxies selected from a compilation of several deep HST 
imaging surveys (GOODS N+S, HUDF & HUDF-P). At M1500 > 
—22 our data are in excellent agreement with the Bouwens et al. 
luminosity function, which is remarkable given the very different 
techniques, data-sets and sky areas utilised by the two studies (the 
largest area in the Bouwens et al. study is the combined GOODS 
N+S fields; <0.1 square degrees). At the brightest magnitudes Fig. 
2a suggests that our sample contains a higher number density of 
galaxies than predicted by the Bouwens et al. luminosity function. 
However, this is not unreasonable given that the large survey area 
of this study provides improved statistics on the bright end of the 
luminosity function, where the Bouwens et al. fit to the luminosity 
function is less well constrained. 

Finally, in Fig. 2a we plot the recent luminosity function fit 
of Iwata et al. (2007) which is based on deep Subaru Suprime-cam 
imaging covering an area of ~ 0.35 square-degrees (HDF-N and 
the J0053+1234 region). Based on cuts in the V — Ic vs Ic — z' 
colour-colour diagram, Iwata et al. (2007) select 853 z ~ 5 LBG 
candidates and derive the luminosity function fit shown as the light 
grey curve in Fig. 2a. It can be seen that our luminosity function es- 
timate is in poor agreement with the Iwata et al. (2007) results, with 
the latter study finding a number density of objects at the brightest 
absolute magnitudes a factor of ~ 3 higher than found here. It is 
difficult to understand how the large numbers of bright LBGs de- 



tected in the Iwata et al. study could be missing from our sample. 
Objects this bright would have very high signal-to-noise photome- 
try in our data-set, and should return the most robust photometric 
redshifts. The Iwata et al. (2007) results are in good agreement with 
their previous estimate of the z ~ 5 luminosity function (Iwata et 
al. 2003) and it has been suggested by several authors (e.g. Ouchi 
et al. 2004a; Yoshida et al. 2006; Bouwens et al. 2007) that the 
selection criteria used in Iwata et al. (2003, 2007) may allow con- 
tamination from low-redshift interlopers. We note that to produce 
number densities at M1500 < —22 as high as reported by Iwata 
et al. (2007), we would have to reinstate into our sample objects 
which have been excluded as suspected ultra-cool galactic stars due 
to their photometry being a poor match to any galaxy SED template 

5.1.2 The z ~ 6 luminosity function 

Previous constraints on the z ~ 6 luminosity function have been 
based, virtually exclusively, on the deep HST imaging available in 
the two GOODS regions, the Hubble Ultra-Deep Field (HUDF) and 
the HUDF parallel fields (e.g. Bunker et al. 2004; Dickinson et al. 
2004; Yan & Windhorst 2004; Malhotra et al. 2005; Bouwens et 
al. 2006, 2007). In Fig. 2b we show the Schechter function fit to 
the z ~ 6 luminosity function derived by Bouwens et al. (2007) 
from a sample of 627 i— dropouts selected from a compilation of 
the deepest available HST imaging Fig. 2b shows that our 

estimate of the z — 6 luminosity function is in reasonable agree- 
ment with the luminosity function fit of Bouwens et al. (2007), al- 
though our number densities are lower by ~ 40% in the region 
around A/1500 — —21, where the Bouwens et al. luminosity func- 
tion fit is still well constrained by data. However, this discrepancy 

5 the reader is also referred to Bouwens et al. (2007) for a detailed com- 
parison of previous constraints on the z ~ 6 luminosity function based on 
HST imaging data 
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Figure 4. The best-fitting Schechter functions to the combined (ground-based+HST) data-sets at z = 5 and z = 6 are shown as the solid curves in the 
left-hand and right-hand panels respectively. During the fitting process the bright end of the luminosity function is constrained using the data from the current 
study (filled circles), while the faint end is constrained using the results of Bouwens et al. (2007), which are based on small area, but ultra-deep HST imaging. 
The best-fitting Schechter function parameters are listed in Table 1. 



is only apparent in comparison to the Schechter function fit derived 
by Bouwens et al., and the actual data from the two studies are in 
excellent agreement (see Fig. 3b). In Fig. 2b we also show the num- 
ber density of objects at M1500 = —21.6 derived by Shimasaku et 
al. (2005), based on a sample of 12 objects selected using imaging 
of the Subaru Deep Field with two medium-band 2 — filters. The 
Shimasaku et al. estimate is a factor of ~ 2 higher than our esti- 
mated number density at M1500 = —21.6. The source of this dis- 
crepancy is not clear. Given the rigorous nature of their selection 
criteria, it seems unlikely that the Shimasaku et al. sample is con- 
taminated by interlopers at the ~ 50% level. However, the survey 
area covered by this work is a factor of ~ 3 larger than that of the 
Shimasaku study, so it is likely that the apparent off-set is simply 
due to a combination of statistical errors and cosmic variance. 



5.2 Combining the ground-based and HST data-sets 

Although the results of the present study provide much improved 
constraints on the bright end of the luminosity function at z — 6 
and 2 = 5, they do not reach faint enough to constrain the nor- 
malisation or faint-end slope of the z > 5 luminosity function. 
Consequently, in this section we explore the possibility of provid- 
ing an improved measurement of the evolution of the luminosity 
function between 2 = 5 and z = 6 by combining our results with 
those of the HST-based study of Bouwens et al. (2007). In Fig. 3 we 
show a comparison of our luminosity function estimates at z = 5 
and 2 = 6 with those of Bouwens et al. (2007). It can be seen that 
in the absolute magnitude range common to both studies the two, 
independent, estimates of the luminosity functions are in excellent 
agreement. Motivated by this agreement, it was decided to combine 
our results with those of Bouwens et al. (2007) to derive Schechter 
function fits to the luminosity function at z = 5 and 2 = 6 based 
on data spanning a factor of ~ 100 in luminosity. 



5.2.1 Fitting the luminosity functions 

The adopted procedure was to use a maximum likelihood technique 
to fit a Schechter function to a finely sampled grid on the apparent 
2'— band magnitude — redshift plane (m z — 2). To fit the luminos- 
ity function at 2 = 5 the grid was populated within the redshift 
interval 4.7 < 2 < 5.3 and the apparent 2'— band magnitude range 
23 < m z < 29. In the magnitude range 23 < m z < 26 the grid 
was populated using the data from this study, while in the magni- 
tude range 26 < m z < 29 the grid was populated according to the 
Schechter function fit to the 2 = 5 luminosity function derived by 
Bouwens et al. (2007). This process was repeated to fit the 2 = 6 
luminosity function, with the exception that the redshift range con- 
sidered was 5.7 < 2 < 6.3. The fitting procedure was to maximise 
the following likelihood function: 

logL = Eijriij logp iy j (4) 

where m t j is the number of galaxies in cell and p(i,j) is the 
probability of finding a galaxy within cell given the choice 
of model parameters. In this context, the probability p(i, j) is natu- 
rally defined as n p (i, j)/$ z , where n p (i,j) is the predicted num- 
ber of galaxies within cell for a given set of model parame- 
ters, and 3> z is the corresponding integrated luminosity function. 

5.2.2 Results 

The best-fitting Schechter functions to the combined ground- 
based+HST data-sets at 2 = 5 and 2 = 6 are shown as the 
solid curves in Fig. 4. The best-fitting Schechter function param- 
eters and their associated uncertainties are reported in Table 1, and 
the la, 2a and 3a confidence regions around the best-fitting val- 
ues of the faint-end slope and characteristic magnitude are plotted 
in Fig. 5. 

It can be seen from Fig. 4 that a Schechter function provides 
a good description of luminosity function at 2 = 5 and 2 = 6 
over the full luminosity range (> 5 magnitudes) sampled by the 
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Figure 5. Constraints on the faint-end slope and characteristic magnitude 
of the luminosity function at z = 5 and z = 6 derived via maximum likeli- 
hood fitting of the combined (ground-based+HST) data-set. The location of 
the best-fitting parameters at each redshift are shown by filled circles, and 
the plotted contours indicate the la, 2a and 3a confidence regions. 



Table 1. The best-fitting Schechter function parameters from our maximum 
likelihood fitting of the combined (ground-based+HST) luminosity function 
data-sets at z = 5 and z = 6. The errors on the faint-end slope and char- 
acteristic magnitude have been derived from the likelihood ratio contours 
shown in Fig. 5. The values of <j>* and their associated errors have been 
calculated by fitting the number counts in each redshift interval. 



Redshift 


</>*/Mpc- 3 


Mfsoo 


a 


5.0 


(9.4 ±1.9) x 10" 4 


-20.73 ±0.11 


-1.66 ±0.06 


6.0 


(1.8 ±0.5) x 10" 3 


-20.04 ±0.12 


-1.71 ±0.11 



combined ground-based+HST data-set. A comparison of the best- 
fitting Schechter function parameters derived here (Table 1) with 
those reported by Bouwens et al. (2007) shows that, at both red- 
shifts, our best-fitting parameters are within the la uncertainties 
reported by the earlier study. With respect to the faint-end slope 
and normalisation this is unsurprising, given that the constraints 
on these two parameters are largely provided by the deeper HST 
imaging data. However, the principle advantage provided by fitting 
the combined data-set is the improved constraints on the value of 
Mi 500 ; particularly at z = 6. Our results suggest that Mi 500 dims 
by 0.69 ± 0.16 magnitudes between z — 5 and 2 = 6. This is 
more dramatic than, although still completely consistent with, the 
dimming of 0.40 ± 0.23 suggested by the Schechter function fits 
derived by Bouwens et al. (2007). With respect to the normalisation 
of the luminosity function, our results suggest that cj>* increases by 
a factor of ~ 2 between z — 5 and 2 = 6, although the compara- 
tively large uncertainty on the normalisation of the Schechter func- 
tion at 2 = 6 means that this evolution is not statistically significant 
(< 2a). As expected, our fit to the combined ground-based+HST 
data-set confirms the Bouwens et al. (2007) result that the faint-end 
slope of the luminosity function at 2 > 5 is steep (a ~ —1.7) and 
shows no sign of evolution in the redshift interval 5 < 2 < 6. 
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Figure 6. 2-point angular correlation function determined for our sample 
of 5.0 < 2 < 6.0 LBGs, with its fitted power-law (solid line). The best- 
fitting parameters for the amplitude and slope are indicated. The dashed 
line represents the power-law fitting only the largest scales measurements 
(6* > 10") assuming a slope fixed to <5 = 0.6. The plot inset presents the x 2 
minimisation to estimate the best-fitting values of and <5, with contours 
showing the la and 3a confidence levels. 



6 CLUSTERING ANALYSIS 

In this section we evaluate the 2-point angular correlation function 
for those LBGs in our sample with primary photometric redshift 
solutions in the interval 5.0 < 2 < 6.0. To measure the angular 
correlation function uj{6) and estimate the related poissonian errors 
we used the Landy & Szalay (1993) estimators. The correlation 
function derived from our 5.0 < 2 < 6.0 LBG sample is shown 
in Fig. 6. The best fit for the angular correlation is assumed to be a 
power-law as in Groth & Peebles (1977): 



= A^O-" - Cs) 



(5) 



with the amplitude at 1 degree A w = 13.0t\\ x 10~ 3 , the slope 
8 = 0.72 ± 0.08, and the integral constraint due to the limited area 
of the survey Cs = 2.34 (determined over the unmasked area). It 
should be noted that because the contamination of our 5 < 2 < 6 
LBG sample from low-redshift interlopers is securely < 10% we 
have made no correction for dilution of the clustering signal. 

In Fig. 6 we also indicate the best-fitting curve determined 
using only the largest scale measurements (6 > 10") and a fixed 
value for the slope of S — 0.6. Based on a sample of 2 ~ 5 LBGs 
selected from the GOODS fields, Lee et al. (2006) find that their an- 
gular clustering measurments show a strong transition on the small- 
est scales (8 < 10"), indicative of a intra/inter dark matter halo 
transition, while on larger scales the angular correlation is well ap- 
proximated by a power-law with a slope of 8 = 0.6. However, as 
can be seen from Fig. 6, our results are consistent with a value of 
8 = 0.6 on large scales, and only present a non-significant devia- 
tion from this slope on the smallest scales (9 = 3.6" corresponding 
to ~ 20/iTOkpc). 
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Figure 7. Postage stamps of the stacked imaging data for the 754 LBGs with primary photometric redshift solutions in the range 5.0 < z < 6.0. From top 
left to bottom right the first eight postage stamps show the stacked BVRi' z' JHK data. The final two postage stamps show the stacked IRAC data at 3.6/im 
and 4.5^m. It can be seen from the second panel that the stacking analysis produces a very faint detection in the V— band (V = 32.3 ± 0.6). This is not 
unexpected given that the Lyman-limit does not pass fully through the Subaru V— band filter until z = 5.5. All the postage stamps have dimensions of 20" X 
20", and are shown on a linear grey-scale in the range ±5<r (where cr is the rms sky noise). The H— band photometry recently became available in UKIDSS 
DR3, and was only used in the stacking analysis. 



In order to compare the clustering of galaxy populations at 
different redshifts it is necessary to derive spatial correlation mea- 
surements. Using the redshift distribution of our sample we can 
derive the correlation length using the relativistic Limber equation 
(Magliocchetti & Maddox 1999) and, in addition, derive a linear 
bias estimation (Magliocchetti et al. 2000) assuming that the dark 
matter behaves as predicted by linear theory. To perform this cal- 
culation we have adopted the redshift distribution for our sample 
as derived from integrating over the redshift probability density 
functions for those objects with primary photometric redshift so- 
lutions in the interval 5.0 < z < 6.0. The correlation length 
can then be evaluated by fixing the slope of the correlation func- 
tion to 7 = 1 + 8 = 1.72. This calculation produces a mea- 
surement of ro = 8-llti's ft^Mpc for the correlation length and 
a value of b = 5.4^J'g for the linear bias. Although we con- 
sider this to be our best estimate of the clustering properties of the 
sample, we note that if we instead adopt the redshift distribution 
based on the best-fitting photometric redshifts alone (z — 5.38; see 
Fig A2) we derive slightly higher, although consistent, values of 
ro = 9.6^i'g h^Mpc and b — 6.2~^\q for the correlation length 
and linear bias respectively. 

It is of interest to compare our clustering results for z < 26.0 
LBGs in the redshift interval 5.0 < z < 6.0 with recent litera- 
ture results for LBG samples with similar limiting magnitudes and 
mean redshifts. Our results are in good agreement with Ouchi et al. 
(2004b) who derived a correlation length of ro = 8.4^2'! /i^Mpc 
for a sample of z' < 25.8 LBGs with a mean redshift of z = 4.7 
in the Subaru Deep Field. In addition, our results are in agreement 
with those of Overzier et al. (2006) who derived a correlation length 
of r = 9.6lg;6 /i^Mpc for a sample of z 85 o < 27.C0LBGs with 
a mean redshift of z = 5.9 in the GOODS fields. Finally, our results 
are also in good agreement with those of Lee et al. (2006), who de- 
rived a value of ro = 7.5_ x 'q fr^Mpc (fixed slope of S = 0.6) 



6 2850 refers to z— band magnitudes obtained through the HST F850LP 
filter, rather than the Subaru z' — filter. 



from their sample of zsso < 26.0 LBGs in the GOODS fields with 
a mean redshift of z = 4.9. 

Based on the predictions of dark matter halo models (Sheth et 
al. 2001 ; Mo & White 2002), and assuming a one-to-one correspon- 
dance between galaxies and halos, we infer that our 5.0 < z < 6.0 
LBGs are likely to be hosted by dark matter halos with masses 
of Mum > 1O 11 ' 5_12 M0, comparable to what is observed for 
L> V LBGs at z ~ 3 - 4. 



7 STACKING ANALYSIS 

In this section we explore the possibility of estimating the typical 
stellar mass of our 5 < z < 6 LBGs from a stacking analysis of the 
available optical+nearlR imaging data. The aim is to test whether 
the typical stellar mass is compatible with the dark matter halo mass 
suggested by the clustering analysis. 

In Fig. 7 we show 20" x 20" postage stamps produced by 
taking a median stack of the available imaging data, centred on 
the positions of the 754 LBG candidates with primary photometric 
redshift solutions in the interval 5.0 < z < 6.0. There are sev- 
eral features of Fig. 7 which are worthy of comment. Firstly, in the 
stack of the B— band photometry the LBG candidates are robustly 
non-detected (B < 33; la), as expected from our initial selection 
criteria. Secondly, it can be seen that there is a faint detection in 
the stacked V— band data. However, the V— band detection is very 
faint (V = 32.3 ± 0.6) and is expected given that the Lyman- 
limit does not pass out of the Subaru Suprime-Cam V— filter until 
z — 5.5. Finally, the two panels on the lower right of Fig. 7 show 
the results of stacking the available Spitzer IRAC data at 3.6/im and 
4.5/im from the SWIRE survey (Lonsdale et al. 2003). Although 
relatively shallow, the SWIRE data is deep enough to produce de- 
tections at 3.6/im and 4.5pim in the median stack. Together with 
the robust photometry from the stacked JHK imaging data, it is 
the extra wavelength coverage provided by the detections at 3.6/im 
and 4.5/im which will allow us to place useful constraints on the 
typical stellar mass of the 5 < z < 6 LBGs. 
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7.1 SED fitting 

Using the same procedure as described in Section 3.1, the best- 
fitting galaxy template to the stacked photometry is shown in 
Fig. 8. It can be seen from the lower panel of Fig. 8 that the fit 
to the stacked photometry provides a robust redshift solution at 



5.43] 



and that any alternative low-redshift solution is se- 



curely ruled out. The best-fitting SED template has an age of 400 
Myrs, consistent with a typical formation redshift of z ~ 7 — 8, 
and no intrinsic reddening (Ay = 0). By considering the range 
of SED templates which provide an acceptable fit to the stacked 
photometry, we calculate that the stellar mass of the LBG stack is 
log 10 (M/M Q ) = 10.0±°:4 (Salpeter IMF). 

7.2 Dark matter to stellar mass ratio 

Although the uncertainties in the determination of the typical stel- 
lar and dark halo mass for the 5 < z < 6 LBG sample are ob- 
viously large, combined with the results of the clustering analysis 
reported in the previous section, the SED fit to the stacked photom- 
etry suggests that the typical dark matter to stellar mass ratio for the 
5 < 2 < 6 LBGs is . A / DM ~ 30 - 100. Given that our LBG sam- 
pie are exclusively drawn from the bright end of the 5 < z < 6 lu- 
minosity function it would seem reasonable to assume that they will 
evolve into objects occupying the high-mass end of the low-redshift 
galaxy stellar mass function (i.e. M > M* tar3 ). In which case, it 
is noteworthy that ^f° M — 50 is in good agreement with the 
measured value for Luminous Red Galaxies (LRGs) in the SDSS 
from galaxy-galaxy lensing (Mandelbaum et al. 2006). Moreover, 
^ D t M — 50 is also in good agreement with the predicted mean 
value for low-redshift LRGs from recent semi-analytic galaxy for- 
mation models (e.g. Almeida et al. 2008) 



8 THE STELLAR MASS FUNCTION 

Although it remains a possibility that a substantial population of 
massive, comparatively red, objects exist at high redshift, it seems 
likely that the luminous (L > L*) LBGs selected in this study are 
amongst the most massive galaxies in existence at 5 < z < 6. As 
such, their number densities and stellar masses have the potential 
to provide constraints on the latest generation of galaxy evolution 
models. 

In Fig. 9 we show an estimate of the galaxy stellar mass func- 
tion at 5 < z < 6 based on combining the luminosity function 
fits presented in Section 5 with the results of the stacking analy- 
sis presented in Section 7. The adopted procedure for estimating 
the stellar mass function is extremely straightforward. Based on 
the Schechter function fits to the luminosity function at z — 5 and 
2 = 6, we have estimated the corresponding stellar mass function 
by simply multiplying the luminosity function fits by the mass-to- 
light ratio derived from our SED fit to the stacked LBG imaging 
data. In Fig. 9 the upper envelope of the grey shaded area is our 
estimate of the stellar mass function at z = 5, while the lower en- 
velope is our corresponding estimate at 2 = 6 (both include the 
lcr uncertainties on the best-fitting Schechter luminosity function 
parameters). For comparison we also plot the stellar mass function 
estimates at z — 5.3 based on the De Lucia & Blaizot (2007) and 
Bower et al. (2006) semi-analytic galaxy formation models (thick 
and thin curves respectively) [j In order to perform a fair compar- 



7 These models are publicly available from the following website: 




A(/Lim) 




"x 40 - 



Figure 8. The upper panel shows the SED fit to the stacked imaging data 
of those LBG candidates with primary photometric redshift solutions in the 
range 5.0 < z < 6.0. In addition to the BV Ri' z' J K photometry used in 
the selection of the LBG sample, the stacked data includes //—band (which 
recently became available in UKIDSS DR3), and detections at 3.6/im and 
4.5^tm from stacking the Spitzer SWIRE data covering the UDS. The bot- 
tom panel shows x 2 versus redshift for the SED fit to the stacked data. As 



expected, the SED fit has a very robust solution at z 
plausible low-redshift solutions are evident. 



5.43 



+0.05 



and no 



ison, we have shifted our stellar mass function estimates to lower 
masses by a factor of 1.8, to account for the fact that the De Lucia & 
Blaizot (2007) and Bower et al. (2006) models are based on IMFs 
which typically return stellar masses a factor of ~ 1.8 lower than 
our Salpeter-based estimates (De Lucia & Blaizot 2007). Finally, 
for reference, the dashed curve in Fig. 9 is the stellar mass function 
at 2 ~ from Cole et al. (2001). 

The mass function predictions from the Bower et al, (2006) 
and De Lucia & Blaizot (2007) models are in good agreement, 
at least qualitatively, with our observational estimate. The gen- 
eral agreement between our estimate, based on the UV-selected lu- 
minosity function, and the model predictions suggests that LBGs 
constitute the majority of the stellar mass density at z > 5 and, 
if the model predictions are correct, a putative population of red- 
dened/older galaxies at 5 < z < 6 does not dominate the stellar 
mass density. Compared to the z ~ mass function, our estimate 
of the high-redshift stellar mass function suggests that the number 
density of ~ 10 10 Mq and ~ 10 11 Mq galaxies in place by z ~ 5 
is ~ 10% and ~ 0.1% of its local value respectively. 



8.1 Stellar mass density 

Given that we have estimated the stellar mass function at z — 5 and 
2 = 6 by simply applying a constant mass-to-light ratio derived 
from our stacking analysis, it is obviously somewhat speculative to 
proceed to estimate the co-moving stellar mass density. 



http://www.g-vo.org/Millennium The model predictions at z = 5.3 are 
adopted because they are the closest available to the mean photometric red- 
shift of our 5 < z < 6 LBG sample (z = 5.38). 
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Figure 9. Estimates of the stellar mass function in the redshift interval 
5 < z < 6. The upper envelope of the grey shaded area is our estimate 
of the stellar mass function at z = 5, and the lower envelope is our cor- 
responding estimate at z = 6 (both include the lcr uncertainties on the 
best-fitting Schechter luminosity function parameters). The thick and thin 
solid lines show the estimated stellar mass function at z = 5.3 derived from 
a combination of the Millennium N-body simulation and the semi-analytic 
galaxy formation models of De Lucia et al. & Blaizot (2007) and Bower et 
al. (2006) respectively. The upper dashed curve is the stellar mass function 
measurement at z ~ from Cole et al. (2001). 



However, for completeness, integrating our stellar mass func- 
tion estimates down to a mass limit of M > 10 9 ' 5 Mq pro- 
vides stellar mass density estimates of ~ 1 x 10 7 Mq Mpc -3 and 
~ 4 x 10 6 Mq Mpc -3 at z — 5 and z = 6 respectively (Salpeter 
IMF). It was decided to integrate our stellar mass function down to 
a limit of M > 10 9 ' 5 Mq for two reasons. Firstly, a mass limit of 
M > 10 9 ' 5 Mq corresponds well to the mass limits of Stark et al. 
(2007) and Yan et al. (2006), who previously estimated the stellar 
mass density at z — 5 and z = 6 respectively. Secondly, given that 
our data-set provides no information on the mass-to-light ratios of 
objects with masses smaller than 10 9 ' 5 Mq, it would clearly be ill 
advised to extrapolate to smaller stellar masses. 

Our stellar mass density estimate at z = 5 is in reasonable 
agreement with the estimate of 6 x 10 6 Mq Mpc -3 derived by 
Stark et al. (2007) from a combined sample of spectroscopic and 
photometrically selected galaxies in the southern GOODS field. 
Moreover, our estimate of the stellar mass density at z — 6 is in 
agreement with the lower limit of 1.1 — 6.7 x 10 6 M Mpc~ 3 de- 
rived by Yan et al. (2006), based on a sample of i— drop galaxies 
in the north and south GOODS fields. The stellar mass densities 
quoted by both Stark et al. (2007) and Yan et al. (2006) are also 
based on a Salpeter IMF. 



9 CONCLUSIONS 

In this paper we have reported the results of a study of a large 
sample of luminous (L > L*) LBGs in the redshift interval 



8 equivalent to integrating the z = 5 and z = 6 luminosity functions to a 
limit of M1500 = —19.3, which involves an extrapolation ~ 1 magnitude 
fainter than the z 1 — band magnitude limit of the UDS/SXDS data set. 



4.7 < z < 6.3. By employing a photometric redshift analysis of 
the available optical+nearlR data we have derived improved esti- 
mates of the bright end of the UV-selected luminosity function at 
z = 5 and z — 6. Moreover, by combining our new results with 
those based on deeper, but small area, HST data we have derived 
improved constraints on the best-fitting Schechter function param- 
eters at 2 = 5 and z = 6. In addition, by studying the angular 
clustering properties of our sample we have determined that lumi- 
nous LBGs at 5 < z < 6 typically lie in dark matter halos with 



masses of 10 



' Mq. Finally, based on the results of a stack- 



ing analysis, we have estimated the galaxy stellar mass functions 
and integrated stellar mass densities at z = 5 and z — 6. Our main 
conclusions can be summarised as follows: 



(i) Our new determination of the bright end of the high-redshift 
luminosity function confirms that significant evolution occurs over 
the redshift interval 5 < z < 6. Based on our results it is clear that 
the luminosity function evolution cannot be described by evolution 
in normalisation (0*) alone, and that some level of evolution in 
Mi 500 is also required. 

(ii) A comparison of our new results with those in the literature 
demonstrates that, within the magnitude range where the two stud- 
ies overlap, our estimates of the luminosity function at z = 5 and 
z = 6 are in excellent agreement with those derived from ultra- 
deep HST imaging data by Bouwens et al. (2007). 

(iii) By combining our estimate of the bright end of the lumi- 
nosity function with the corresponding estimates of the faint end 
by Bouwens et al. (2007), it is possible to fit the luminosity func- 
tion at z — 5 and z — 6 over a luminosity range spanning a fac- 
tor of ~ 100. Based on this combined ground-based+HST data-set 
we find the following best-fitting Schechter function parameters: 
M? 60 o = -20.73 ± 0.11, 0* = 0.0009 ± 0.0002 Mpc" 3 and 
a — —1.66 ± 0.06 for the luminosity function at z = 5, and 
Misoo = -20.04 ± 0.12, 0* = 0.0018 ± 0.0005 Mpc" 3 and 
a = -1.71 ±0.11 atz = 6. 

(iv) These results are consistent with the corresponding 
Schechter function parameters derived by Bouwens et al. (2007) al- 
though, due to the improved statistics at the bright end provided by 
our wide survey area, the fits to the combined ground-based+HST 
data-set provide improved constraints on the evolution of Mi 500 in 
particular. 

(v) An analysis of their angular clustering properties demon- 
strates that luminous L > L* LBGs at 5 < z < 6 are strongly clus- 
tered, with a correlation length of ro = 8.1 j^'J ft^Mpc. Compari- 
son of these clustering results with theoretical models suggests that 
these LBGs typically reside in dark matter halos with masses of 
~ 10 n - 5 - 12 M Q . 

(vi) An SED fit to a stack of the available optical+NearlR data 
for our sample suggests that luminous LBGs at 5 < z < 6 have 
typical stellar masses of ~ 10 10 Mq. Combined with the results 
of the clustering analysis this suggests that the typical ratio of 
dark matter to stellar mass for luminous LBGs at 5 < z < 6 is 
#2M_ ~ 30 - 100. 

(vii) Assuming that the mass-to-light ratio derived from the SED 
fit to the stacked LBG imaging data is representative, we use our 
best-fitting Schechter function parameters for the z — 5 and z — 6 
luminosity functions to estimate the corresponding stellar mass 
functions. Although clearly subject to large uncertainties, our stel- 
lar mass function estimates are consistent with the latest predic- 
tions of the semi-analytic galaxy formation models of De Lucia & 
Blaizot (2007) and Bower et al. (2006). 



© 0000 RAS, MNRAS 000, 000-000 



12 R.J. McLure et al. 



(viii) Based on our stellar mass function estimates we calculate 
that the stellar mass in place at z = 5 and z = 6 is ~ 1 x 10 7 Mq 
Mpc -3 and ~4x 10 6 Mq Mpc -3 respectively. Although uncer- 
tain, these independent estimates of the integrated stellar mass den- 
sity are consistent with the results of studies utilising the ultra-deep 
Spitzer IRAC data in the GOODS fields (Stark et al. 2007; Yan et 
al. 2006). 
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APPENDIX A: LUMINOSITY FUNCTION ESTIMATES: 
TECHNICAL DETAILS 

When deriving our estimates of the LBG luminosity functions at 
z = 5 and z — 6 in Section 4 we employed several techniques 
which differ somewhat from those which have been commonly 
adopted in previous literature studies. Consequently, it is perhaps 
instructive to investigate what influence these techniques have on 
the derived luminosity function estimates. In this section we first 
investigate the influence of our completeness corrections, before 
proceeding to investigate the influence of our decision to make use 
of the full photometric redshift probability density function for each 
LBG candidate. 



Al Correction factor 

As previously discussed in Section 4, to estimate the LBG lu- 
minosity function at z — 5 and z = 6 we used the V/V m ax 
maximum likelihood estimator of Schmidt (1968), adapted to in- 
corporate the photometric redshift probability density function of 
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each LBG candidate. Consequently, within a given redshift inter- 
val (Zmin < z < z max ), our estimate of the luminosity function 
within a given absolute magnitude bin was: 



4>{M) = J2 



^ (mi) Ci(m 

zi (mi) 



z')Pi(z')Sz' 



V l max (m l ,z') 



(Al) 



where the summation runs over the full sample, and z\ — ► Z2 is the 
redshift range within which a candidate's absolute magnitude lies 
within the magnitude bin in question. As described in detail in Sec- 
tion 4, the quantity Ci(rrii, z) is a correction factor derived from 
realistic monte-carlo simultions of the evolving LBG population, 
and accounts for sample incompleteness, object blending and con- 
tamination from objects photometrically scattered into the sample 
from faint- ward of the 2'— band magnitude limit. 

To illustrate the influence of the correction factor we show in 
Fig Al our estimates of the z = 5 and z = 6 LBG luminosity 
functions with and without the inclusion of Cj(mi, z). In Fig Al 
the open and filled black symbols are the estimates of the z — 5 and 
z — 6 luminosity functions including the correction factor (iden- 
tical to the estimates displayed in Fig 1), while the grey symbols 
are the equivalent estimates setting dim, z) — 1. It can be seen 
from Fig Al that the correction factor does not have a significant 
impact on the shape of the luminosity function estimates. At the 
bright end, where our sample is most complete, the off-set between 
the black and grey data-points is simply due to the ~ 15% correc- 
tion for objects lost due to blending in the crowded Subaru images. 
In the faintest absolute magnitude bins the inclusion of the correc- 
tion factor boosts the number densities by a roughly constant factor 
of ~ 30% (0.1 dex). This effect is due to a balance between the in- 
creasing positive correction that has to be made for incompleteness 
as the data approach the z = 26 magnitude limit, and the increas- 
ingly negative correction that has to be made for objects photomet- 
rically scattered into the sample from faint-ward of the magnitude 
limit. 



A2 Comparing different photometric redshift techniques 

When deriving our estimates of the LBG luminosity functions at 
z — 5 and z — 6 in Section 4 we used the full photometric redshift 
probability density functions provided by our SED template fitting 
code in order to make full use of the available information. In this 
section we investigate the influence of this technique on the derived 
luminosity function estimates. Of particular interest is to investigate 
how the derived luminosity function estimates change if we simply 
treat the best-fitting photometric redshift for each LBG as unique, 
and ignore the possibility of secondary redshift solutions. 



A2. 1 Redshift distributions 

The first straightforward comparison of the two techniques is to 
compare the estimated redshift distributions. In Fig A2 we show 
the LBG redshift distributions in the three redshift intervals utilised 
in this paper: 4.7 < z < 5.3, 5.0 < z < 6.0 and 5.7 < z < 6.3. 
The redshift distributions shown in the left-hand column are de- 
rived by treating the best-fitting photometric redshift for each LBG 
as unique and robust. The redshift distributions shown in the right- 
hand column are derived by integrating over the full redshift proba- 
bility density functions as done in Section 4. The number of objects 
included in each redshift distribution is listed in the top right-hand 
corner of each panel. 

Although the redshift distributions derived via both techniques 
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Figure Al. Estimates of the bright end of the UV-selected luminosity func- 
tion at z = 5 and z = 6. The open and filled black data-points are derived 
using Eqn Al, including the correction factor Cj(mj,z), and are identi- 
cal to those presented in Fig 1. The grey data-points are the corresponding 
estimates derived by setting C,(mj, z)=\ (see text for discussion). 
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Figure A2. A comparison of the LBG redshift distributions as derived us- 
ing the best-fitting photometric redshifts (left-hand column) and integrating 
over the full redshift probability density function (right-hand column). The 
appropriate redshift range and the number of objects lying within that range 
are listed in the top right-hand corner of each panel. 
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are broadly similar, it can be seen from Fig A2 that they do differ in 
detail. Perhaps the most noticeable difference is that adopting the 
best-fitting photometric redshifts alone predicts ~ 50% more ob- 
jects in the highest redshift interval (5.7 < z < 6.3). Armed with 
this information we now proceed to investigate how the differences 
in redshift distributions affect the estimated luminosity functions. 

A2.2 Luminosity function estimates 

In Fig A3 we show estimates for the bright end of the z — 5 and 
z = 6 LBG luminosity functions. The left-hand panels show the 
estimates derived in Section 4 by adopting the photometric redshift 
probability density function for each LBG. The right-hand panels 
show the luminosity function estimates derived from simply adopt- 
ing the best-fitting photometric redshift for each source. In each 
panel we also plot the maximum likelihood Schechter function fits 
to the LBG luminosity function at either z — 5 or z = 6 as derived 
in Section 5.2. 

It can be seen from Fig A3 that the estimate of the z = 5 lu- 
minosity function is virtually identical using both techniques. This 
is as expected given the similarity of the 4.7 < z < 5.3 redshift 
distributions shown in the top panel of Fig A2, and demonstrates 
that the photometric redshifts at z ~ 5 are sufficiently robust that 
both techniques lead to indistinguishable conclusions. In contrast, 
although the two luminosity function estimates at z = 6 are clearly 
consistent, there are differences in detail. Specifically, it can be seen 
from Fig A3 that the luminosity function estimate based on the 
best-fitting photometric redshifts predicts more objects in the three 
faintest absolute magnitude bins. This effect is entirely consistent 
with the differences between the 5.7 < z < 6.3 redshift distribu- 
tions shown in Fig A2, which demonstrate that using the best-fitting 
photometric redshifts alone predicts ~ 50% more objects at z ~ 6. 

In conclusion, it is clear from the information displayed in 
Figs A2 & A3 that the only substantive difference between the 
two techniques is for the faintest LBGs at z ~ 6. This is per- 
haps to be expected, given that these are the very objects for which 
the best-fitting photometric redshifts are least robust, and was the 
primary motivation for exploiting the extra information contained 
within the full redshift probability density function for each ob- 
ject. However, the good agreement between the luminosity func- 
tion estimate shown in the bottom right panel of Fig A3 and the 
best-fitting Schechter function derived in Section 5.2 confirms that 
adopting either technique would lead to the same conclusions re- 
garding the form, and evolution, of the LBG luminosity function 
between z = 5 and z = 6. 
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Figure A3. A comparison of the LBG luminosity function estimates at 
2 = 5 and 2 = 6 resulting from using the full redshift probability density 
function for each object (left-hand plots) and the best-fitting photometric 
redshifts alone (right-hand plots). The top two plots show the different lu- 
minosity function estimates at z = 5, while the bottom two plots show the 
different luminosity function estimates at z = 6. The maximum likelihood 
fits to the 2 = 5 and 2 = 6 luminosity functions derived in Section 5.2 are 
plotted in the top and bottom panels respectively. 
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